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ABSTRACT 


This  research  memorandum  explains 
the  origin  of  the  VCNO's  tasking  of  CNA 
to  study  sustaining  engineering.  It 
describes  the  analytic  and  procedural 
approach  to  the  problem  as  originally 
conceived  and  reports  some  of  the  early 
substantive  findings.  The  most  important 
of  these  was  that  the  Navy  does  not 
know,  nor  can  it  currently  find  out  in  a 
comprehensive  and  systematic  way,  the 
specific  services  it  receives  as  a 
result  of  its  annual  expenditure  of 
approximately  $1  billion  on  sustaining 
engineering  for  aircraft  and  missiles. 
When  these  preliminary  results  were 
briefed  to  the  VCNO,  the  original  task¬ 
ing  was  revised  and  expanded.  The  gen¬ 
eral  contents  of  subsequent  documents  in 
this  series  are  also  outlined. 


INTRODUCTION 


It  is  generally  accepted  that  there  are  three  basic  phases  in  wea¬ 
pon  system  engineering:  original  design  engineering,  sustaining  engi¬ 
neering,  and  systems  engineering.  While  different  nomenclature  is  some¬ 
times  used,  and  "systems  engineering"  can  have  other  meanings  in  other 
contexts,  most  groups  involved  in  weapon  system  engineering  agree  with 
the  following  definitions  developed  by  the  Center  for  Naval  Analyses 
(CNA ) : 


•  Original  Design  Engineering.  All  engineering  effort, 
including  that  involved  in  development,  test,  and  evalua¬ 
tion,  leading  up  to  a  final  approved  weapon  system  design; 
that  is,  all  engineering  effort  occurring  prior  to  comple¬ 
tion  of  the  Physical  Configuration  Audit,  when  the  design 
is  frozen  for  the  first  full-scale  production  article. 

•  Sustaining  Engineering.  Engineering  effort  involved  in 
setting  up  and  improving  production  processes,  ensuring 
systems  integration,  advising  on  and  correcting  deficien¬ 
cies  discovered  during  manufacturing  and  service  use,  and 
assisting  in  other  phases  of  manufacturing  such  as  quality 
assurance  and  configuration  control;  also  the  engineering 
required  to  ensure  that  current  production  articles  meet 
explicit  or  implicit  specifications. 

•  Systems  Engineering.  Engineering  effort  applied  to  exist¬ 
ing  weapon  systems  to  enhance  performance,  reliability,  or 
maintainability  significantly  beyond  current  specifica¬ 
tions;  or,  engineering  involved  in  the  design  of  improved 
models  of  a  current  weapon  system. 

For  reasons  to  be  explained  subsequently,  in  June  1985  the  then 
Vice  Chief  of  Naval  Operations  (VCNO),  Admiral  Ronald  Hays,  tasked  CNA 
to  study  sustaining  engineering  (SE).  This  research  memorandum  is  the 
first  of  a  series  reporting  the  results  of  the  study.  It  is  intended  to 
serve  several  purposes: 

•  Explain  the  origin  of  the  VCNO's  tasking  and  the  perceived 
nature  of  the  problem. 

•  Outline  the  methodology,  the  plan  of  action  as  originally 
conceived,  and  the  changes  made  as  the  study  progressed. 

•  Report  certain  of  the  findings  presented  to  the  VCNO  in  a 
briefing  in  April  1986. 

•  Summarize  the  new  and  expanded  VCNO  tasking  resulting  from 
the  April  briefing. 


•  Describe  actions  taken  by  the  Naval  Air  Systems  Command 
(NAVAIR)  as  a  result  of  the  April  briefing. 

•  Preview  the  research  memorandums  that  will  follow  this 
one . 

ORIGIN  OF  THE  VCNO’s  TASKING 

In  May  1985,  CNA  presented  a  briefing  to  Admiral  Hays,  then  the 
VCNO,  and  Mr.  Paisley,  then  the  Assistant  Secretary  of  the  Navy 
(Research,  Engineering,  and  Systems),  on  ways  to  improve  budgeting, 
accounting,  and  controls  for  what  was  called  Contractor  Support  Services 
(CSS)  at  that  time  and  is  n^w  referred  to  as  Contracted  Advisory  and 
Assistance  Services  (CAAS).  One  of  the  major  reforms  proposed  was  to 
adopt  a  new  set  of  sound  and  consistent  definitions  for  CAAS  as  a  whole 
and  for  each  of  its  major  categories.  A  principal  criterion  in  develop¬ 
ing  the  overall  definition  of  CAAS  was  to  emphasize  that  it  involves 
temporary  or  intermittent  "consulting  services"  of  an  "advisory  and 
assistance"  nature.  Use  of  these  criteria  suggested  that  engineering 
and  technical  services  integral  to  the  normal  development  and  production 
processes  ( i . e . ,  the  three  basic  phases  of  weapon  system  engineering 
defined  above)  were  not  CAAS  and  the  formal  definitions  in  the  new  CAAS 
directives  clearly  reflected  these  exclusions. 

While  agreeing  with  these  definitional  distinctions,  Admiral  Hays 
and  ASN  Paisley  shared  certain  concerns  about  SE.  First,  it  was  the 
least  understood  aspect  of  weapon  system  engineering.  Second,  the  new 
definitions  put  it  beyond  the  purview  of  the  proposed  new  controls  being 
applied  to  CAAS.  Third,  it  provided  a  potential  "hiding  place"  for  one 
of  the  other  major  nonengineering  categories  of  CAAS — Stud ies,  Analyses, 
and  Evaluations.  And,  fourth,  evidence  developed  during  the  CAAS  study 
indicated  that  under  certain  contractual  arrangements,  the  Navy  has  lit¬ 
tle  visibility  into  or  control  over  what  weapon  system  contractors  do 
with  the  man-hours  and  funds  proposed  for  SE.  After  the  briefing,  the 
VCNO  charged  CNA  to  address  these  issues.  This  verbal  charge  was  for¬ 
malized  in  a  VCNO  tasking  memorandum,  which  is  reproduced  in  appendix  A. 

THE  PROBLEM  AS  ORIGINALLY  CONCEIVED 

As  originally  conceived,  analyzing  the  problem  aDpeared  to  require 
undertaking  the  following  tasks: 

•  Define  SE  and  the  related  forms  of  weapon  system  engineer¬ 
ing  in  a  comprehensive  and  integrated  way. 


’ .  CNA's  recommended  changes  were  subsequently  incorporated  in  a  Depart¬ 
ment  of  Defense  (DOD)  directive  and  companion  SECNAVINST.  Both  of  these 
directives  use  the  new  term  "Contractec  Advisory  and  Assistance  Ser¬ 
vices"  (CAAS)  in  lieu  of  the  older  "Contractor  Support  Services"  (CSS). 
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•  Define  SE,  in  particular,  in  terms  of  the  specific  tasks 
or  functions  performed. 

•  Estimate  on  a  consistent  basis  the  dollars  invested  in  SE 
and  aggregate  them  in  various  ways,  particularly  by  weapon 
system  and  system  command. 

•  Determine  the  products  or  services  obtained  from  SE 
expenditures . 

•  Categorize  the  products  or  services  obtained  on  the  basis 
of  their  value  to  the  Navy. 

•  Suggest  alternative  methods  for  increasing  the  Navy's  vis¬ 
ibility  into  and  control  over  SE  to  ensure  it  gets  its 
money's  worth. 

•  Determine  if  a  more  uniform  contractor  cost  reporting  sys¬ 
tem  would  be  feasible  and  cost  effective. 

As  this  first  phase  of  the  study  progressed,  however,  it  became 
necessary  to  modify  the  original  plan.  The  changes,  and  the  reasons 
therefor,  are  described  in  subsequent  sections  of  this  report. 

METHODOLOGY 

A  fortunate  early  development  in  the  study  was  the  interest  and 
support  expressed  by  the  F-14  program  office  in  NAVAIR  ( PMA-24 1 ) .  Dis¬ 
cussions  with  members  of  that  office  confirmed  and  expanded  the  study 
team's  original  ideas  about  what  constitutes  weapon  system  engineering 
in  general  and  SE  in  particular.  PMA-24 1  staff  members  also  acquainted 
the  study  team  with  a  number  of  current  issues  in  estimating,  managing, 
and  accounting  for  SE  hours  and  dollars.  They  also  expressed  general 
agreement  with  what  the  study  team  thought  were  important  "drivers"  of 
SE.  In  addition,  they  agreed  with  the  hypothesis  that,  without  pertur¬ 
bations  in  the  production  process  or  a  sharply  increased  requirement  to 
support  operational  aircraft,  the  need  for  SE  in  a  particular  program 
should  decline  over  time. 

Similar  discussions  were  conducted  with  program  managers,  business 
managers,  contracting  officers,  and  engineers  in  NAVAIR  program  offices. 
As  a  result,  the  study  team  was  able  to  develop  definitions  of  the  three 
basic  phases  of  weapon  system  engineering  (outlined  in  the  Introduc¬ 
tion),  the  specific  tasks  constituting  SE  (shown  in  table  1),  a  formal 
questionnaire  (appendix  B),  and  a  standard  data  format  (appendix  C). 

At  this  point,  four  significant  decisions  were  made.  First,  the 
basic  form  of  the  study  would  be  a  comparative  analysis  among  weapon 
systems  in  three  broad  groups--aircraft  and  missiles,  shipboard  weapons 
and  related  systems,  and  surface  ships  and  submarines.  Second,  because 
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5KS  OF  SUSTAIN  INC  ENCI NEE- INu 


’  .  Participating  ir.  laying  out  and  modifying  procuet.or.  proc 

2.  Resolving  problems  of  systems  integration 

3.  Correcting  production  problems  identified  ty : 

Production  supervisors  anc  quality  assurance  perscrr.e. 
Contractor  and  Service  fiignt  tests  (Flight  CRABS' 

4.  Undertaking  engineering  investigations  resulting  from  f.eet- 
ldentified  problems: 

Accidents  and  incidents 

Unsatisfactory  reports  by  squadron  pilots  (design-related 
problems  rather  than  simple  malfunctions) 

5-  Preparing  ail  engineering  change  proposals  (ECPs) 

6.  Preparing  detailed  drawings  and  specifications  for  approved 
correction-of-defects  (COD)  engineering  change  proposals  (ECPs) 

7.  Supervising  implementation  of  COD  ECP  work 

8.  Conducting  engineering  investigations  to  enhance  producibil ity 

9.  Initiating  and  evaluating  value  engineering  proposals 

10.  Conducting  engineering  efforts  to  ensure  that  reliability  and 
maintainability  meet  contract  specification  or  desired  levels 

11.  Providing  engineering  input  to  configuration  control 

12.  Updating  drawings,  rewriting  specifications,  and  providing 
engineering  input  to  technical  publications  and  manuals 

13.  Maintaining  contractual  engineering  documentation 

14.  Overseeing  supplier  operations  and  products 

15.  Maintaining  liaison  with  Service  acquisition  managers  and  their 
field  activities 

16.  Managing  in-house  sustaining  engineering  efforts 
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of  the  study  team  members'  experience,  and  because  of  the  continuing 
support  of  PMA-241,  the  study  team  chose  the  aircraft  and  missiles  group 
as  the  first  to  be  analyzed.  Third,  because  the  data  resources  of  the 
relevant  Navy  program  offices  were  limited,  and  because  a  first-hand 
appreciation  of  the  environment  in  which  SE  work  is  done  was  important, 
on-site  interviews  and  facility  tours  of  contractors  were  added  to  the 
schedule.  And  fourth,  to  enrich  the  data  base  and  get  different  views 
about  managing  SE,  Air  Force  aircraft  were  added  to  the  systems  to  be 
studied. 

The  sites  visited  and  weapon  systems  studied  in  this  first  phase  of 
the  analysis  were:  the  Air  Force's  F-16  System  Program  Office  (SPO)  at 
Wr ight-Patterson  AFB,  Ohio;  Grumman  Aerospace  Corp.  at  Bethpage, 

New  York  ( F- 1 4 ) ;  McDonnell  Douglas  Aircraft  Corp.  at  St.  Louis,  Missouri 
(F-15  and  F-18);  General  Dynamics  Corp.  at  Fort  Worth,  Texas  (F-16); 
Texas  Instruments  at  Lewisville,  Texas  (HARM  missile);  Lockheed  Missile 
and  Space  Co.  at  Sunnyvale,  California  (Trident  C-4  missile);  and  Boeing 
Aerospace  Corp.  at  Seattle,  Washington  (727  and  737  commercial  aircraft 
and  the  E-3/AWACS  military  aircraft). 

PRELIMINARY  RESULTS  BRIEFED  TO  THE  VCNO 

On  29  April  1986  the  results  of  the  study  up  to  that  point  were 
briefed  to  Admiral  James  Busey ,  the  current  VCNO  (and  former  Commander, 
NAVAIR),  Mr.  Paisley  (ASN-RE&S),  and  a  large  group  of  concerned  senior 
Navy  officers.  The  briefing  provided  only  preliminary  results,  not  just 
because  of  the  small  number  of  aircraft  and  missiles  studied,  but  also 
because  of  certain  limitations  affecting  the  study  to  that  point.  These 
limitations  are  described  in  the  following  discussion  of  the  preliminary 
findings. 

Despite  the  limitations,  a  number  of  the  preliminary  results  could 
be  stated  with  some  confidence: 

•  There  was  near-unanimous  agreement  about  the  definitions 
of  the  three  basic  phases  of  weapon  system  engineering  and 
the  individual  tasks  constituting  SE.  Not  all  of  the  con¬ 
tractors  perform  all  of  these  tasks  (table  1),  but  they 
agreed  that  conceptually  the  tasks  are  part  of  the  overall 
SE  function. 

•  The  Navy  spends  roughly  $1  billion  annually  on  SE  for  air¬ 
craft  and  missiles.  These  dollars  buy  approximately 

17  million  SE  man-hours.  For  reasons  noted  below,  how¬ 
ever,  it  is  impossible  to  say  on  any  consistent  or  compre¬ 
hensive  basis  what  outputs  (products  or  services)  these 
inputs  buy. 

•  Methods  for  estimating,  budgeting,  recording,  and  control¬ 
ling  SE  hours  and  dollars  varied  widely  among  contractors. 


Even  in  the  best  case,  only  a  partial  accounting  for  the 
products  and  services  could  be  made--at  least  in  terms  of 
the  tasks  detailed  in  CNA's  definition  of  SE.  In  the 
worst  case,  no  accounting  was  possible  below  the  aggregate 
of  all  SE  hours  and  dollars.  There  were  also  wide  differ¬ 
ences  in  how,  and  how  well,  the  estimating  and  budgeting 
functions  were  performed. 

Similarly,  there  were  wide  differences  among  program 
offices  within  NAVAIR  in  their  ability  to  identify, 
account  for,  and  analyze  even  the  aggregate  SE  hour  and 
cost  data.  The  HARM  program  was  a  notable  exception;  it 
has  an  excellent  tracking  and  analysis  system. 

Obtaining  complete  and  consistent  data,  even  on  aggregate 
SE  hours  and  costs,  was  a  major  problem  for  the  study 
team.  While  contractors,  Plant  Representative  Offices 
(PROs),  Navy  program  offices,  and  Air  Force  System  Program 
Offices  (SPOs)  agreed  with  CNA's  definitions  and  found  the 
standard  quantitative  data  sheet  (appendix  C)  a  sensible 
way  to  array  the  data,  it  required  varying  degrees  of 
manipulation  of  the  contractors'  internal  accounting  sys¬ 
tems  to  put  the  relevant  data  in  CNA's  format.  In  fact, 
because  the  data  set  was  still  incomplete  and  contained 
unresolved  inconsistencies  at  the  time  of  the  April  brief¬ 
ing,  very  little  quantitative  analysis  could  be  performed. 
However,  because  even  the  limited  amount  done  was  provoca¬ 
tive,  the  VCNO  asked  the  study  team  to  resolve  the  data 
deficiencies  and  complete  the  analyses.  Over  the  course 
of  the  next  several  months  this  was  done.  The  results 
will  be  reported  in  later  research  memorandums  in  this 
series . 

A  uniform  system  for  reporting  contractor  cost  and  man¬ 
hour  data  is  specified  in  DOD  Instruction  7000.11.  The 
instruction  is,  however,  widely  ignored,  even  when  the 
reports  it  calls  for  are  submitted.  The  problem  seems  to 
be  that  it  was  designed  for  certain  kinds  of  macro  cost 
modeling  rather  than  for  management  purposes  or  the  kinds 
of  analyses  CNA  undertakes.  (Some  contractors  use  parts 
of  the  Contractor  Cost  Data  Reporting  (CCDR)  system  and 
its  associated  Work  Breakdown  Structure  (WBS)  for  their 
own  internal  tracking  and  controlling  systems.)  The  CCDR' 
WBS  system  is,  however,  virtually  useless  for  managing  SE 
because  SE  hours  are  scattered  throughout  the  WBS.  Fur¬ 
ther,  the  time  lag  in  CCDR  reporting  precludes  its  use  for 
real-time  management  or  analysis.  Even  as  a  basis  for 
long-term  analyses,  it  has  shortcomings  because  of  anoma¬ 
lies  in  the  WBS.  For  example,  the  WBS  is  a  mixture  of 
physical  parts  of  the  aircraft  and  groups  of  functions 


(e.g.,  systems  engineering),  some  of  which  contain  dissim¬ 
ilar  components.  A  reform  is  needed  and  appears  feasible, 
but  it  would  require  a  large-scale,  multiservice  effort. 

•  Methods  of  accounting  for  and  controlling  Class  I  Engi¬ 
neering  Change  Proposals  (i.e.,  those  that  result  in  some 
change  to  form,  fit,  function,  or  cost  of  a  system)  also 
varied  much  more  than  would  have  been  expected  for  such  a 
long  established  and  ostensibly  closely  regulated  process. 

•  It  gradually  became  evident  during  this  first  phase  of  the 
study  that  there  are  substantial  differences  in  how  super¬ 
visory  groups  (contractors'  program  management  offices  and 
engineering  departments,  PROs,  Defense  Contract  Audit 
Agency  offices,  Navy  program  offices,  and  Air  Force  SPOs) 
are  organized  and  staffed.  These  differences  exist  not 
only  among  contractors  and  between  the  Navy  and  the 

Air  Force,  but  even  within  the  Navy.  For  example,  the 
Strategic  Systems  Project  Office  (SSPO)  and  the  Joint 
Cruise  Missile  Project  Office  (JCMPO)  in  the  Navy  are  far 
more  heavily  staffed  and  more  self-sufficiently  organized 
than  any  other  Navy  program  office — and,  in  fact,  they 
outdo  the  largest  Air  Force  SPOs  in  these  respects.  Other 
less  dramatic,  but  nevertheless  significant,  differences 
exist  within  NAVAIR. 

The  study  team's  observations  on  this  point  are  not 
an  endorsement  of  "more  is  better"  when  it  comes  to 
management  staffing.  Too  many  offices  at  the  same  level 
doing  closely  related  functions,  or  too  many  levels  of 
management,  are  common  organizational  weaknesses.  But  in 
the  present  case,  larger  commitments  of  oversight 
resources  at  the  program  office  and  SPO  levels  do  seem  to 
have  a  positive  effect,  not  only  on  SE  management  but  on 
overall  management  and  product  quality  as  well.  The  unre¬ 
solved  issues  are  these:  At  what  point  are  the  marginal 
benefits  of  investments  in  oversight  personnel  and  systems 
no  longer  worth  the  marginal  costs?  And  even  before  that 
point  is  reached,  what  are  the  opportunity  costs  of  the 
management  manpower  committed  to  SE? 

•  As  the  first  phase  of  the  study  progressed,  the  inability 
of  the  study  team  (or  the  Navy,  or  even  the  contractors) 
to  identify  what  services  resulted  from  the  SE  man-hours 
expended  became  increasingly  evident.  The  research  focus 
shifted,  therefore,  from  its  original  emphasis  on  measur¬ 
ing  and  comparing  outputs  to  concern  about  SE's  contribu¬ 
tion  to  product  qual ity--that  is,  to  measures  of  effec¬ 
tiveness  related  to  aircraft  safety,  reliability,  and 
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maintainability  which  are  believed  to  be  influenced  by  SE. 
This  shift  occurred  too  late  to  permit  systematic  gather¬ 
ing  of  data  in  time  for  the  April  briefing  for  measures 
such  as  not-mission-capable  rates,  zero-defect  aircraft, 
unscheduled  maintenance  hours  per  flight  hour,  mean  time 
between  failures,  and  aircraft  mishap  rates  caused  by 
materiel  and  engineering  shortcomings.  What  data  did 
become  available  suggested  that  well  controlled  and  moder¬ 
ate  use  of  SE  hours  and  dollars  (on  a  comparative  basis) 
did  not  degrade  product  quality.  Clearly,  however,  this 
new  focus  on  SE  usage  in  relation  to  weapon  system  quality 
warranted  further  conceptual  development  and  more  complete 
and  systematic  data  gathering. 

•  Many  alternatives  exist  for  improving  management  of  SE. 
Identifying  preferred  mixes  is  beyond  the  scope  of  this 
paper.  Nevertheless,  it  appears  that  implementing  some  of 
these  alternatives — including  more  regular  purchase  of 
full  cost  and  schedule  information,  greater  use  of  incen¬ 
tive  features  in  contracts,  and  better  methods  for  estima¬ 
ting  SE  needs — would  strengthen  the  negotiating  position 
of  Navy  program  managers.  Given  the  magnitude  of  the 
expenditures  involved  (approximately  $1  billion  annually), 
even  modest  improvements  could  significantly  reduce  the 
cost  of  Navy  aircraft  and  missiles.  (NAVAIR  has  already 
implemented  some  of  the  study  team’s  recommendations.) 

THE  VCNO's  REVISED  AND  EXPANDED  TASKING 

As  a  result  of  the  CNA  briefing  of  29  April  1986  to  Admiral  Busey 
and  ASN  Paisley,  the  VCNO  revised  and  expanded  CNA's  tasking.  (See 
appendix  E  for  the  new  tasking  memorandum.)  The  changes  fall  under  two 
main  headings:  scope  and  emphasis. 

Scope 

The  VCNO  directed  CNA  to: 

•  Solve  the  data  problems  and  complete  the  work  on  the  orig¬ 
inal  sample  of  aircraft  and  missiles 


•  Add  the  A-6E,  E-2C,  AV-8B,  and  P-3  aircraft  to  the  sample, 
as  well  as  the  Tomahawk,  Harpoon,  Phoenix,  Sparrow,  and 
Sidewinder  missiles 

•  Begin  work  on  surface  ships,  submarines,  and  shipboard 
weapon  systems,  including  torpedoes  and  electronic 
subsystems . 


The  VCNO  also  directed  CNA  to  emphasize  the  following  topics: 

•  Relationships  between  SE  usage,  its  management,  and  mea¬ 
sures  of  effectiveness  related  to  product  quality 

•  Techniques  for  estimating  the  need  for  SE 

•  Management  of  Engineering  Change  Proposals 

•  Organizational  arrangements  to  enhance  program  managers' 
ability  to  negotiate  and  control  SE. 

NAVAIR's  ACTIONS 

Immediately  following  CNA's  briefing  to  the  VCNO,  The  Commander  of 
NAVAIR,  Vice  Admiral  Wilkinson,  formed  an  ad  hoc  SE  study  group  to  cri¬ 
tique  CNA's  preliminary  results  and  make  recommendations  to  the  Command 
Advisory  Board.  This  group  completed  its  work  within  three  months. 

While  the  members  of  the  study  group  registered  a  few  dissents  from 
the  CNA  proposals,  in  general  they  agreed  with  CNA's  findings  and  sug¬ 
gestions.  More  importantly,  the  Board  recommended,  and  Vice  Admiral 
Wilkinson  approved,  a  plan  of  action  to  accomplish  the  following: 

•  Formally  define  SE  and  the  tasks  under  it  that  may  be 
properly  charged  to  production  contracts. 

•  Develop  and  promulgate  a  uniform  policy  on  handling 
Class  I  Engineering  Change  Proposals. 

•  Adopt,  as  a  command-wide  model,  the  procedures  used  by  the 
HARM  missile  procurement  contracting  officer  (PCO)  for 
managing  SE.  (CNA  had  identified  this  PCO  as  being  the 
most  informed  and  effective  one  encountered  in  the  inter¬ 
viewing  and  data  gathering  process.) 

•  Bring  the  class  desk  officers  more  fully  into  the  contract 
negotiating  process. 

•  Create  a  command-wide  historical  data  base  on  SE. 
ADDITIONAL  RESEARCH  MEMORANDUMS 

The  second  research  memorandum  in  this  series  will  cover  the 
results  of  site  visits  involving  the  F-14,  F - 1 5 ,  F— 1 6 ,  F— 1 8 ,  and  AWACS 
aircraft  and  HARM  and  Trident  C-4  missiles.  Subsequent  ones  will  cover 
site  visits  involving  the  additional  aircraft  and  missiles  specified  by 
the  VCNO.  Still  others  will  summarize  the  analyses  pertaining  to  air¬ 
craft  and  review  the  supporting  data. 
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MEMORANDUM  FDR  DC  PRESIDENT,  COTTER  FOR  NAVAL  ANALYSES 
SUbj:  ANALYSIS  OF  SUSTAINING  ENGINEERING 

1.  Ccnwdore  J.  0.  Taylor  presented  a  briefing  recently  on  our  progress  in 
fop  roving  the  aanageaent  and  reporting  of  Contracted  Advisory  and  Assistance 
Services  (CAAS).  My  assessoent  is  that  a  great  deal  has  been  accoeplished, 
both  conceptually  and  practically,  in  bringing  this  long-standing  problem 
under  control.  I  appreciate  the  active  and  efficient  role  CNA  has  played  in 
contributing  to  these  iaproveoents. 

2.  In  responding  to  an  earlier  concern  of  the  Assistant  Secretary  of  the  Navy 
(REIS).  Coaaodore  Taylor  verified  that  an  iapertant  element  of  the  engineering 
effort  that  goes  into  weapon  system  production  and  deployment— sustaining 
engineering— is,  under  some  contractual  arrangements,  apparently  iemune  to  the 
controls  fa  the 40*  CAAS  aanageaeRt  system.  A  case  can  be  aede  that  several 
hundred  afllion  dollars’  worth  of  engineering  effort  annually  is  available  to 
contractors  to  use  as  they  choose  with  little  cr  no  visibility  or  control  fcy 
senior  Navy  aanageaent. 

3.  We  aust  pursue  iaaediately  a  range  of  actions  to  gain  better  control  of 
.sustaining  engineering  costs.  One  of  these  actions  «ist  be  a  concentrated. 

systematic  study  to  deteraine  the  extent  and  severity  of  the  preplan  and  to 
propose  and  evaluate  a  broad  range  of  al:ema:iv«  solutions.  Your  involvement 
in  the  "parent”  CAAS/CSS  preplan  over  the  past  year  ideally  qualifies  CKA  to 
undertake  this  study.  The  study  should  address.  Put  need  not  be  liaited  to, 
the  following  questions: 

a.  How  auch  are?**  spending  annually  for  sustaining  engineering? 

P.  What  are  we  buying? 

c.  What  do  we  need? 

d.  Whet  can  we  do  to  iaprove  our  oanageaent  and  control  of  sustaining 
engineering  costs? 

4.  Ccaaodort  Taylor  discovered  also  that  a  aajor  hindrance  to  cooperative 
analysis  In  this  area  is  the  wide  difference  in  the  systcas  used  by 
contractors  for  recording  contract  cost  data.  In  the  course  of  the  study, 
therefore,  I  would  like  yoe  to  investigate  the  feasibility  of  a  eore  uni  fora 
contract  cost  reporting  syst«. 
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Sutoj:  ANALYSIS  OF  SUSTAINING  ENGINEERING 

S.  Pleaso  provide  an  analysis  plan  to  accomplish  t has*  tasks  at  /our  earliest 
convenience.  I  wold  appreciate  jour  preliminary  results  in  September  1985 
with  a  final  report  by  1  December  198S.  NY  point  of  contact  for  this  project 
is  Commodore  J.  0.  Taylor,  OP-921,  G95-32S3. 
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APPENDIX  B 


QUESTIONNAIRE  ON  MANAGEMENT  OF  SUSTAINING  ENGINEERING 

(This  questionnaire  was  sent  to  contractors  in  advance  of  the  study 
team's  visit  to  assist  them  in  preparing  for  the  visit  and  to  provide  a 
structure  for  the  on-site  discussions.  Most  contractors  had  formal 
responses  ready  at  the  time  of  the  visit.) 


1.  Given  the  attached  definitions  of  "original  design  engineering," 
"sustaining  engineering,"  and  "systems  engineering,"  are  there  any 
significant  overlaps?  Are  there  any  other  major  categories  of  engi¬ 
neering  besides  these  three? 

2.  In  terms  of  the  kinds  of  contracts  involved  (e.g.,  FFP,  FPI,  CPFF), 
what  has  been  the  contractual  history  of  this  system  in  the  produc¬ 
tion  phase?  Is  sustaining  engineering  (SE)  included  in  the  current 
production  contract?  Is  it  a  contract  line  item?  Is  it  in  a  separ¬ 
ate  contract?  If  it  is  in  a  separate  contract,  what  type  of  con¬ 
tract  is  it?  Can  you  break  SE  man-hours  out  of  the  relevant  con¬ 
tracts  by  all  or  some  of  the  components  shown  in  the  attachment?  If 
not,  can  they  be  broken  out  using  some  other  method?  In  either 
case,  can  you  supply  these  data  on  an  actual- basis  for  all  of  the 
regular  production  years,  plus  estimates  for  FY  1985  and  FY  1986? 
(See  attached  standard  data  format.)  [The  standard  data  format  is 
presented  in  appendix  C  of  this  research  memorandum. ] 

3.  Can  you  show  how  sustaining  engineering  man-hours  are  a  function  of 
any  of  the  following  SE  "workload  indicators?" 

Production  rates 

Class  I  Engineering  Change  Proposals  (ECPs) 

Class  II  engineering  changes 

Pre-acceptance  tests 

Material  review  actions 

Vendor  certification  and  recertification 

Number  of  deployed  systems 

Engineering  investigations 

Contract  data  items 

Amount  of  test  equipment 

Field  service  reports 

Message  traffic  from  fleet 

Other  correspondence 

Do  you  use  these  or  similar  indicators  as  the  basis  for  calculating 
the  total  number  of  SE  man-hours  included  in  your  annual  contract 
proposals?  Or,  do  you  budget  SE  on  a  level-of-effort  basis  geared 
to  an  anticipated  peak  demand?  Or,  do  you  negotiate  and  authorize 
specific  SE  tasks  either  sequentially  or  as  an  annual  package?  Or 
do  you  estimate  SE  man-hour  requirements  based  on  a  "learning  curve" 


derived  from  earlier  production  history  of  this  or  a  similar  sys¬ 
tem?  If  you  use  none  of  these  methods,  how  do  you  determine  SE 
needs? 


4.  It  seems  reasonable  to  assume  that  the  number  of  sustaining  engi¬ 
neering  man-hours  in  support  of  production  would  decrease  over  time 
(absent  design  changes),  or  as  production  quantities  are  reduced. 

Is  this  correct?  If  not,  what  are  the  factors  that  would  counter¬ 
act  these  assumed  trends? 

5.  It  also  seems  reasonable  to  assume  that  problems  associated  with 
in-service  weapon  systems,  and  hence,  the  sustaining  engineering 
effort  needed  to  resolve  them,  would  decrease  over  time.  Is  this 
correct?  If  not,  what  countervailing  factors  are  at  work? 

6.  Are  the  following  included  as  separate  line  items  in  an  overall 
procurement  contract,  or  are  they  covered  by  separate  contracts? 
Which  of  these  categories  include  sustaining  engineering? 

A.  Production  of  the  basic  weapon  system 

B.  Initial  spares 

C.  Special  support  equipment 

D.  Trainers 


E.  Technical  publications 

F.  ILS 

7.  What  is  the  extent  of  competition  in  subcontracting/vendor 
procurement? 

8.  Is  there  any  foreign  subcontracting  or  co-production?  How  exten¬ 
sive  is  it? 

9.  For  recent  FYs,  by  fiscal  year,  how  many  Class  I  Correction  of 
Deficiency  (COD)  ECPs  were  prepared;  of  those  prepared,  how  many 
were  implemented?  For  all  ECPs  prepared  in  each  fiscal  year,  how 
many  man-hours  were  required  and  of  these  totals  how  many  were  con¬ 
sidered  SE?  For  those  ECPs  implemented  in  each  fiscal  year,  how 
many  man-hours  were  required  and  of  those  how  many  were  considered 
SE? 

10.  How  are  engineering  man-hours  for  correction  of  deficiency  problems 
charged?  By  what  system  are  design  correction  costs  assigned  to  a 
particular  fiscal  year  contract(s)?  How  are  corrections  while 
under  warranty  paid  for  (i.e.,  implicitly  or  explicitly  in  the  pro¬ 
duction  contracts)? 
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11.  Do  annual  production  contracts  include  costs  for  correcting  nonwar¬ 
ranty  design  deficiencies  that  affect  all  weapons  systems  in  ser¬ 
vice?  If  so,  wouldn't  this  distort  the  apparent  production  cost 
and  price  of  the  weapons  systems  in  that  year's  contract?  Would  it 
be  better  to  have  a  separate  contract  or  a  separate  line  item  in 
the  production  contract  for  each  of  those  costs  not  associated  with 
production  of  the  new  weapons  systems? 

12.  Where  and  how  are  the  costs  of  answering  "what  if"  questions 
charged? 

13.  Does  your  sustaining  engineering  staff  produce  formal  studies?  If 
so,  could  you  furnish  a  list  of  these  for  the  past  few  years?  How 
is  the  time  charged  to  produce  these  studies?  Who  determines  what 
studies  will  be  produced? 

14.  How  much,  if  any,  "systems  engineering"  work  is  done  under  produc¬ 
tion  contracts  (i.e.,  engineering  effort  to  improve  the  performance 
of  the  weapon  system  beyond  the  current  design)?  Where  would  these 
hours  be  charged? 

15.  Do  you  report  hours  and  dollars  in  accordance  with  DOD  Instruction 
7000.11  (i.e.,  CCDR/WBS  data)? 

16.  In  addition  to,  or  in  lieu  of,  CCDR  data  do  you  gather  and  report 
Cost  and  Schedule  Control  System  (C/SCS)  information?  If  so,  what 
is  the  annual  cost  to  the  government?  If  it  is  used  only  for 
internal  company  purposes,  what  would  it  cost  the  government  to 
obtain  it? 


SUSTAINING  ENGINEERING  STANDARD  DATA  FORMAT 
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APPENDIX  D 


PROCEDURAL  LESSONS  LEARNED 


In  addition  to  the  substantive  findings  outlined  in  the  body  of 
this  working  paper,  the  study  team  learned  (and  re-learned!)  some  proce¬ 
dural  lessons  that  may  prove  helpful  to  others  who  undertake  similar 
kinds  of  studies. 

•  Allow  time  for  unexpected  delays.  For  example,  key  let¬ 
ters  were  lost  (including  the  original  copy  of  Admiral 
Hays'  first  tasking  memorandum)  or  were  misdirected.  And, 
in  fact,  expect  everything  to  be  harder  to  accomplish  than 
it  seems  and  to  take  longer  than  one  might  think. 

•  Preparing  for  and  following  up  on  site  visits  are  incredi¬ 
bly  time  consuming — but  absolutely  necessary.  Giving  con¬ 
tractors  the  maximum  lead  time  creates  good  will  and  bet¬ 
ter  responses.  Thank-you  letters  mentioning  particularly 
helpful  individuals  smooth  the  way  for  follow-up  calls  or 
visits. 

•  Work  through  the  staff. of  the  relevant  Plant  Representa¬ 
tive's  Office  (PRO)  until  it  suggests  direct  contacts  with 
contractor  staff. 

•  Allowing  the  PRO  staff  and  the  contractor  to  review  the 
draft  of  trip  reports  builds  confidence  and  helps  ensure 
accuracy . 

•  Be  candid,  but  respect  confidences.  Do  everything  neces¬ 
sary  to  protect  competition-sensitive  information. 

•  Air  Force  personnel  were  extremely  cooperative.  Where 
they  have  something  to  offer— insights ,  experience,  data, 
a  different  viewpoint  or  modus  operand i--make  use  of  it. 
(Aircraft  and  missiles,  of  course,  were  natural  subjects 
for  this  kind  of  cross-fertilization.)  Be  sure  here,  too, 
that  particularly  helpful  individuals  are  formally 
recognized . 

•  In  cases  such  as  this  one,  in  which  the  sponsor  of  the 
study  is  a  senior  Navy  official,  having  a  flag-rank  point 
of  contact  and  a  task  coordinator  who  is  knowledgeable  and 
supportive  can  greatly  facilitate  progress. 
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DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  CHIEF  OF  NAVAL  OPERATIONS 
WASHINGTON.  OC  20350  2000 


IN  ACFLV  were*  *0 

Ser  09/ 6U3Q1045 
23  May  1986 


MEMORANDUM  FOR  THE  PRESIDENT,  CENTER  OF  NAVAL  ANALYSES 
Sub j :  ANALYSIS  OF  SUSTAINING  ENGINEERING 

1.  On  Tuesday,  29  April,  CNA  briefed  me  on  the  preliminary 
results  of  their  sustaining  engineering  analysis.  The  original 
tasking  requested  CNA  to  address  the  following  questions: 

a.  How  much  are  we  spending  annually  for  sustaining 
engineering? 

b.  What  are  we  buying? 

c.  What  do  we  need? 

d.  What  can  we  do  to  improve  our  management  and  control  of 
sustaining  engineering  costs? 

e.  Is  a  more  uniform  contract  cost  reporting  system  feasible? 

2.  The  briefing  provided  valuable  insight  and  partial  answers  to 
these  questions  based  on  an  examination  of  the  F-14,  F-15,  F-16, 
F/A-18,  and  AWACS  aircraft,  and  the  HARM  and  C-4  missiles. 

Because  of  the  potential  benefits  from  better  management  and 
control  of  sustaining  engineering,  I  would  like  CNA  to  continue 
the  analysis  by  undertaking  the  following  tasks: 

a.  Develop  consistent  and  complete  data  on  the  original 
sample  of  aircraft  and  missiles  in  order  to  provide  a  detailed 
understanding  of  sustaining  engineering. 

b.  Expand  the  sample  of  aircraft  and  missiles  to  include  the 
A-6,  E-2,  AV-8B,  P-3,  Tomahawk,  Harpoon,  Phoenix,  Sparrow,  and 
Sidewinder. 

c.  In  order  to  improve  the  management  and  control  of 
sustaining  engineering,  attempt  to  establish  relationships  between 
amounts  of  sustaining  engineering  used,  methods  of  management,  and 
Measures  of  Effectiveness  (MOEs)  related  to  system  reliability/ 
maintainability/ safety. 


d.  To  the  extent  practicable,  develop  standards  and  methods 
for  estimating  future  sustaining  engineering  requirements  on  an 
annual  per  unit  basis. 


e.  In  conjunction  with  subparagraphs  a.  through  d.  above, 
include  a  review  of  the  control  and  funding  of  engineering  change 
proposals  with  recommendations. 
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Sub j i  ANALYSIS  OF  SUSTAINING  ENGINEERING 


f.  Additionally,  if  evidence  warrants,  suggest  organizational 
arrangements  that  will  enhance  program  managers'  ability  to 
negotiate  and  control  sustaining  engineering  in  production 
contracts . 


g.  Adapt  and  extend  the  study's  methodology  and  scope  to 
include  surface  ships,  submarines,  and  ship-based  weapon  systems 
to  include  torpedoes  and  major  electronics  subsystems. 

3.  Please  provide  an  analysis  plan  to  accomplish  these  tasks  at 
your  earliest  convenience.  I  would  appreciate  a  progress  report 
in  September  1986,  and  a  final  briefing  by  December  1986.  My 
point  of  contact  for  this  project  remains  Rear  Admiral  J.  D. 
Taylor,  OP-921,  695-3262. 


Copy  to* 

ASN  ( S&L) 

ASN  (RE&S) 
COMNAVSEASYSCOM 
COMNAVAIRSYSCOM 
COMS  PAWARS YSCOM 
CNO  (OP-090) 
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J.  B.  BUSEY  /  , 
ADMIRAL.  U.S.  NAVY/ 
VICE  CHIEF  OF  NAV.«L 


